In this paper, the authors propose a technique called Noisy Embedding Instruction Fine Tuning (NEFTune) to improve the performance of fine-tuning generative language models (LLMs) on instruction datasets. They show that by adding random noise to the embedding vectors of the training data during fine-tuning, they can achieve significant improvements in conversational quality without additional computational or data overhead.

The authors first introduce the importance of instruction fine-tuning for LLMs and how it can enhance their usefulness. They highlight that traditional instruction fine-tuning methods focus on cross-task generalization and may not fully capture the ability of LLMs to provide free-form answers to open-ended queries. They then discuss related work in the field.

The NEFTune technique involves adding random noise vectors to the embedding vectors of the training data during the forward pass of fine-tuning. The noise is generated by sampling independent and identically distributed (iid) entries from a uniform distribution in the range [-1, 1]. The entire noise vector is then scaled by a factor of alpha divided by the square root of the sequence length multiplied by the embedding dimension. This scaling rule is borrowed from adversarial machine learning literature and results in a random vector with an expected Euclidean magnitude of approximately alpha divided by the square root of 3.

To evaluate the effectiveness of NEFTune, the authors conduct experiments using different datasets and LLM models. They measure the performance of the models using the AlpacaEval benchmark, which compares Text-Davinci-003 generations to model generations. They find that NEFTune significantly improves the performance of LLMs on conversational tasks, with performance boosts of up to 35 percentage points observed. They also evaluate the models on the OpenLLM Leaderboard tasks to ensure that NEFTune does not negatively impact model capabilities in other areas.

The authors analyze the results of their experiments and provide insights into the mechanisms behind the success of NEFTune. They propose that NEFTune reduces overfitting to the specifics of the instruction dataset, leading to better generalization and increased coherence in the generated responses. They also investigate the relationship between response length and token diversity and find that NEFTune does not compromise token diversity despite generating longer responses.

To further validate their findings, the authors conduct a human study and observe that human annotators prefer NEFTune-generated responses in the majority of cases. They also compare NEFTune models against standard finetuned models using the AlpacaEval benchmark and find that NEFTune consistently outperforms the standard models.

In conclusion, the authors demonstrate that NEFTune is a simple yet effective technique for improving the performance of fine-tuned LLMs on instruction datasets. The noise added during fine-tuning reduces overfitting and enhances conversational quality without sacrificing token diversity. NEFTune provides a valuable regularization approach for LLM training and highlights the importance of algorithms and regularizers in improving LLM performance.

Words: 464